a viewpoint detection program step for detecting the location/posture yfa 
viewpoint of the operator; 

an inputting program step for inputting a real space image^rresponding to 
the location/posture of a viewpoint of the operator; 

a geometric information acquisition program^step for acquiring geometric 
information of real object(s); 

a recognition program step for recognizing a current relative relationship 
between the virtual object(s) and real otyect(s); 

a rule memory for storing rules for controlling the action of the virtual 

object(s); 

a computation program step for determining the next action of the virtual 
object(s) in accordance'with the rules stored in said rule memory and in correspondence 
with the location/pasture of the real object(s), and computing the location/posture of the 
virtual object(s > )after the determined action; and 

/ 

a presentation program step for generating at least one image of the virtual 
objection the basis of the location/posture of the virtual object(s) after the action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed 
reality space to the operator by superimposing the image(s) of the virtual object(s) on the 
/operator's view of the real space. 

REMARKS 

Applicants request favorable reconsideration and allowance of this 
application in view of the foregoing amendments and the following remarks. 
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Claims 1-39 are pending in this application, with Claims 1, 20, and 39 being 

independent. 

Claims 1-4, 7, 9-11, 17-20, and 39 have been amended. Applicants submit 
that support for these amendments can be found in the original disclosure, and therefore no 
new matter has been added. 

Claims 1-11, 14, 15, 17-19, 20-30, 33, 34 and 36-39 were rejected under 35 
U.S.C. § 102(b) as being clearly anticipated by U.S. Patent No. 5,577,981 (Jarvik). Claims 
12, 13, 16, 31, 32, and 35 are rejected under 35 U.S.C. § 103(a) as being unpatentable over 
Jarvik . Applicants respectfully traverse these rejections for the reasons discussed below. 

As recited in independent Claims 1, 20, and 39, the present invention relates 
to a simulator with which an operator plays a game with virtual object(s) in a mixed reality 
space. A mixed reality space combines both virtual images and real space images. 
Accordingly, the present invention recited in Claims 1, 20,and 39 includes, inter alia, the 
feature of inputting a real space image corresponding to a location/posture of a viewpoint 
of an operator. Applicants submit that the cited art fails to disclose or suggest at least this 
feature. 

In particular, Applicants submit that Jarvik is essentially directed to a virtual 
reality exercise machine for a single user. Since the exercise machine is directed to virtual 
reality (VR) and not to mixed reality (MR), i.e., Jarvik is not directed to a combination of 
virtual images and real space images, Jarvik does not disclose or suggest at least the 
feature of inputting a real space image corresponding to a location/posture of a viewpoint 
of a user. Accordingly, Applicants submit that the present invention recited in Claims 1, 
20, and 39 is patentable over the cited art. 
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The dependent claims are patentable over the cited art for at least the same 
reasons as the independent claims, as well as for the additional features they recite. 

For the foregoing reasons, Applicants submit that this application is in 
condition for allowance. Favorable reconsideration, withdrawal of the outstanding 
rejections, and an early Notice of Allowance are requested. 

Applicants' undersigned attorney may be reached in our Washington, D.C. 
office by telephone at (202) 530-1010. All correspondence should continue to be directed 
to our below-listed address. 



Respectfully submitted, 

Attorney for Applicants 
Brian L. Klock 
Registration No. 36,570 



FITZPATRICK, CELLA, HARPER & SCINTO 
30 Rockefeller Plaza 
New York, New York 101 12-3801 
Facsimile: (212)218-2200 

BLKAlmj 
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VERSION WITH MARKINGS TO SHOW CHANGES MADE TO THE CLAIMS 

1 . (Twice Amended) A simulator apparatus with which an operator plays a 
simulation with virtual object(s) in mixed reality space including a virtual space and real space, 
said simulator comprising: 

a viewpoint detection [means for detecting] unit adapted to detect the 
location/posture of a viewpoint of the operator; 

an inputting unit adapted to input a real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition [means for acquiring] unit adapted to 
recognize geometric information of real object(s); 

a recognition [means for recognizing] unit adapted to recognize a current relative 
relationship between the virtual object(s) and real object(s); 

a rule memory [for storing] adapted to store rules for controlling the action of the 
virtual object(s); 

a computation [means for determining] unit adapted to determine the next action 
of the virtual object(s) in accordance with the rules stored in said rule memory and in 
correspondence with the location/posture of the real object(s), and [computing] to compute the 
location/posture of the virtual object(s) after the determined action; and 

a presentation [means for generating] unit adapted to generate at least one image 
of the virtual object(s) on the basis of the location/posture of the virtual object(s) after the 
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determined action and the location/posture of the viewpoint position of the operator, and [for 
representing] to represent the mixed reality space to the operator by superimposing the image(s) 
of the virtual object(s) on the operator's view of the real space. 

2. (Amended) The apparatus according to claim 1, wherein said presentation umt 
[means] further comprises: 

an image-capturing unit adapted to capture [means for capturing] real space 
images of said operator's view of real space images of said player's view of the real space; 

an image generation unit adapted to generate [means for generating] mixed reality 
images representing the mixed reality space by superimposing or overlaying said image(s) of the 
virtual object(s) on said real space images; and 

a video see-through type display [means] that the operator wears wherein said 
mixed reality images are displayed. 

3. (Amended) The apparatus according to claim 1, wherein said presentation umt 
[means] further comprises an optical see-through type display [means] that the operator wears 
wherein said virtual object image(s) are displayed. 

4. (Amended) The apparatus according to claim 1, further comprising, 

a status detector that detects a [detecting means for detecting] status of the 



operator; 



Application No.: 09/506,084 
Attorney Docket No.: 02355.01 1 106 

wherein said computation unit [means] determines a next action of the virtual 
object in accordance with the rule stored in said rule memory and in correspondence with the 
location/posture of the real object and/or the status of the operator, and computes [computing] a 
location/posture of the virtual object after the determined action. 

7. (Amended) The apparatus according to claim 1 , wherein the real object 
includes the operator himself or herself, and said recognition unit [means] recognizes a current 
relative relationship between the virtual object and the operator. 

9. (Amended) The apparatus according to claim 1, wherein the real object is an 
object which is fixed in position, and 

said geometric information acquisition unit [means] comprises: 

a predetermined memory for pre-storing location information and shape 

information of the real object; and 

a reading unit that reads [means for reading] out the location information and 

shape information of the real object from said predetermined memory as needed. 

10. (Amended) The apparatus according to claim 1, wherein the real object is an 
object which is movable but does not deform, and 

said geometric information acquisition unit [means] comprises: 
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a predetermined memory for pre-storing shape information of the real 

object; 

a location/posture sensor for detecting a location/posture of the real object; 

and 

a setting unit that sets [means for setting] a region the real object is 
expected to occupy in the mixed real space in accordance with the detected 
location/posture of the real object. 

1 1 . (Amended) The apparatus according to claim 1, wherein the real object is an 

operator, and 

said geometric information acquisition unit [means] comprises: 

a sensor for detecting a location/posture of a head of the operator; and 
a setting unit that sets [means for setting] a region having a fixed, known 

shape that approximates the operator in the mixed reality space in accordance with 

the detected location/posture of the operator's head. 

17. (Amended) The apparatus according to claim 1 1, wherein when the operator 
is one of the real objects, an output from said viewpoint detection unit [means] for detecting the 
location/posture of the viewpoint of the operator is also used as information which is to be 
acquired by said geometric information acquisition unit [means] and pertains to a location and 
shape of the operator. 
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18. (Amended) The apparatus according to claim 1, wherein said viewpoint 
detection unit [means] detects a location/posture of the head of the operator, and 

said apparatus further comprises a detector that detects [detection means for 
detecting] a location/posture of a hand of the operator; and 

a recognition unit adapted to recognize [means for recognizing] a relative location 
of the hand of the operator with respect to the head as a command on the basis of an output from 
said detector [detection means]. 

19. (Amended) The apparatus according to claim 1, wherein said presentation 
unit [means] comprises: 

an alignment unit that aligns [means for aligning] the location/posture of the real 
object to the location/posture of the virtual object after movement; 

a generation unit that generates [means for generating] an image of the virtual 
object after alignment in correspondence with an occlusion relationship; and 

a head-mounted display device. 



20. (Twice Amended) An image processing method for a simulator apparatus 
with which an operator plays a game with virtual object(s) in a mixed reality space comprising: 
a viewpoint detection step for detecting the location/posture of a viewpoint of the 

operator; 
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an inputting step for inputting a real space imaee corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition step for acquiring geometric information of 

real object(s); 

a recognition step for recognizing a current relative relationship between the 
virtual object(s) and real object(s); 

a computation step for determining the next action of the virtual object(s) in 
accordance with the rules stored in a rule memory, which stores rules for controlling the action of 
the virtual object(s), and in correspondence with the location/posture of the real object(s), and 
computing the location/posture of the virtual object(s) after the determined action; and 

a presentation step for generating at least one image of the virtual object(s) on the 
basis of the location/posture of the virtual object(s) after the determined action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed reality 
space to the operator by superimposing the image(s) of virtual object(s) on the operator's view of 
the real space. 

39. (Twice Amended) A storage medium which stores a program of an image 
processing method for a simulator apparatus with which an operator plays a simulation with 
virtual object(s) in a mixed reality space including a virtual space and real space comprising: 

a viewpoint detection program step for detecting the location/posture of a 
viewpoint of the operator; 
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an inputting program step for inputting a real space image corresponding to the 
location/posture of a viewpoint of the operator; 

a geometric information acquisition program step for acquiring geometric 
information of real object(s); 

a recognition program step for recognizing a current relative relationship between 
. . the -virtual object(s) and real object(s); - ~ " " " " 

a rule memory for storing rules for controlling the action of the virtual object(s); 

a computation program step for determining the next action of the virtual object(s) 
in accordance with the rules stored in said rule memory and in correspondence with the 
location/posture of the real object(s), and computing the location/posture of the virtual object(s) 
after the determined action; and 

a presentation program step for generating at least one image of the virtual object 
on the basis of the location/posture of the virtual object(s) after the action and the 
location/posture of the viewpoint position of the operator, and for representing the mixed reality 
space to the operator by superimposing the image(s) of the virtual object(s) on the operator's 
view of the real space. 

DC_MAIN 110136 vl 
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